• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    optimizations are provided for the processing of final stage reprojection for a multilayered scene. a scene is generated, which is based on a predicted pose of a part of a computer system. a sub-region is identified within one of the layers and is isolated from the other regions in the scene. thereafter, the final stage reprojection processing is applied to this sub-region selectively / unlike other regions in the scene that do not undergo the same late state reprojection processing.
    公开号:BR112019016952A2
    申请号:R112019016952
    申请日:2018-03-20
    公开日:2020-04-14
    发明作者:James Muff Adam；Leung Gene；Powers Bradford Jeffrey；Comparan Miguel；Scott Haraden Ryan；Ozguner Tolga
    申请人:Microsoft Technology Licensing Llc；
    IPC主号:
    专利说明:

    Invention Patent Descriptive Report for SELECTIVE APPLICATION AND REPROJECTION PROCESSING IN LAYER SUB-REGIONS TO IMPROVE FINAL STAGE REPROJECTION ENERGY.
    BACKGROUND OF THE INVENTION [0001] Computers and computing systems impacted almost every aspect of modern life. Computers are generally involved in work, recreation, health, transportation, entertainment, home management, etc.
    [0002] Mixed reality computer systems, which include virtual reality systems and augmented reality systems, have recently received significant interest in their ability to create immersive experiences for users. Conventional augmented reality systems create an augmented reality scenario by visually presenting virtual objects in the real world. In contrast, conventional virtual reality systems create a more immersive experience so that an entire view of the user is obstructed by a virtual world. As used here, mixed reality, augmented reality and virtual reality systems are described and referred to interchangeably. Unless specifically established or unless specifically required, as understood by someone skilled in the art, the descriptions here apply equally to any type of mixed reality system, including augmented reality systems, virtual reality systems and / or any other similar system capable of displaying virtual objects to a user.
    [0003] Mixed reality computer systems typically use one or more devices on the body (for example, a head-mounted device, a portable device, etc.). The head-mounted device provides a viewfinder, sometimes referred to as the head-mounted viewfinder (hereinafter HMD), which allows the user to
    Petition 870190079091, of 08/15/2019, p. 21/76
    2/40 rio visualize integrated and / or superimposed visual information in the user's environment. For example, a mixed reality system can present visual information to the user, through the HMD, in the form of a simulated object on a real table surface.
    [0004] Continuous advances in hardware capabilities and rendering technologies have greatly increased the realism of virtual objects and scenes displayed to a user within a mixed reality environment. For example, in mixed reality environments, virtual objects can be placed inside the real world in order to give the impression that the virtual object is part of the real world. When a user moves within the real world, the mixed reality environment updates automatically so that the user is provided with the appropriate perspective and view of the virtual object. This mixed reality environment is often referred to as a computer-generated scene, or simply a scene.
    [0005] In such systems, the user's body (specifically the user's head and corresponding HMD) can move in real time in relation to the virtual environment. For example, in a mixed reality application, if the user tilts his head in one direction, he would not expect the image or hologram to tilt with him. Ideally, the system would measure the user's head position and render images at a very fast rate to eliminate any turbulence and deviation in the image position, as perceived by the user. However, typical graphics processing units (GPU) currently render frames at only 30 to 60 frames per second, depending on the quality and performance of the GPU. This results in a potential delay of 16 to 33 milliseconds between the point in time when the head position is detected and when the image is actually displayed on the HMD. Additional latency can also be associated with the time that is required to determine the
    Petition 870190079091, of 08/15/2019, p. 22/76
    3/40 new head position and / or delays between the GPU frame buffer and the final adjusted display. The result is a potentially large error between where the user would expect an image and where the image is actually displayed, thereby degrading the user experience. In some cases, the user may also experience discomfort and disorientation when the virtual images are not displayed in the appropriate locations, particularly during the user's head movement and HMD.
    [0006] In an effort to reduce or eliminate some of the preceding rendering errors, existing systems apply final stage corrections to make final adjustments to the image after the image rendered by the GPU. This process is performed before the pixels are displayed in order to compensate for the last rotation, translation, and / or enlargements resulting from the user's head movement. This adjustment process is often referred to as Late State Adjustment, Final Stage Reprojection, LSR or LSR Adjustments. Hereafter, this invention will use the abbreviation LSR. Since frames can be rendered at a high rate with high resolution, existing systems that employ LSR may require a large amount of DRAM broadband and power. It will be appreciated that, in the context of a battery-powered and wireless HMD, the chip size as well as the bandwidth and power requirements can be very important, which can add to the challenges and difficulties associated with reality scenes. mixed for one user.
    [0007] As suggested above, many mixed reality computer systems are wireless battery powered devices that suffer from operational power restrictions. These restrictions are designed to extend the life of the battery so that the user can enjoy more time with the scene. However, many computer system operations significantly impact battery life
    Petition 870190079091, of 08/15/2019, p. 23/76
    4/40 laugh at the computer system. For example, carrying out data acquisition operations and LSR actions are prime examples of such battery charging operations. Consequently, there is a strong need in the field to efficiently improve the energy consumption of systems that perform these types of operations.
    [0008] The subject claimed here is not limited to modalities that solve any disadvantages or that operate only in environments such as those described above. Instead, this background is provided to illustrate only an area of exemplary technology where some of the modalities described here can be practiced.
    BRIEF SUMMARY OF THE INVENTION [0009] The modalities described here relate to systems, hardware storage devices, and methods that optimize LSR processing.
    [0010] Some modalities generate a computer generated scene that includes several layers. This scene is generated based on a predicted first pose from a part of a computer system (for example, an HMD or other mixed reality device). The modalities also make a sub-region within one or more layers to be identified. This sub-region is distinguished from at least one other region in the one or more layers. So, the modalities make this subregion isolated from other region (s). Subsequently, the modalities cause LSR processing to be applied in the isolated subregion while refraining from applying LSR processing, or at least the same LSR processing, to the other region (s).
    [0011] This Summary is provided to introduce a selection of concepts in a simplified form which is further described below in the Detailed Description. This Summary is not intended to identify ca
    Petition 870190079091, of 08/15/2019, p. 24/76
    5/40 key features or essential features of the claimed subject, nor is it intended to be used as an aid in determining the scope of the claimed subject.
    [0012] Additional features and advantages will be presented in the description below, and in part will be obvious from the description, or can be learned by practicing the teachings contained here. Features and advantages of the invention can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. Features of the present invention will become completely evident from the following description and appended claims, or can be learned by practice as presented later here
    BRIEF DESCRIPTION OF THE DRAWINGS [0013] In order to describe the way in which the other features and advantages mentioned above can be obtained, a more particular description of the subject briefly described above will be made by reference to the specific modalities that are illustrated in the attached drawings. The understanding of these drawings represents only typical modalities and, therefore, they should not be considered limiting in scope, modalities will be described and explained with additional specificity and detail through the use of the attached drawings in which:
    [0014] Figure 1 illustrates a basic architectural view of a computer system that is capable of carrying out the described processes and methods.
    [0015] Figure 2 illustrates another modality of a computer system that is capable of carrying out the described processes and methods.
    [0016] Figure 3 illustrates a high-level overview of a multilayered scene.
    [0017] Figure 4 provides a realistic representation of a layer that can be used in a multilayered scene.
    Petition 870190079091, of 08/15/2019, p. 25/76
    6/40 [0018] Figure 5 provides another realistic representation of a layer that can be used in a multilayered scene.
    [0019] Figure 6 illustrates a rendering composed of multiple layers.
    [0020] Figure 7 illustrates another rendering composed of multiple layers.
    [0021] Figure 8 illustrates an example of how a pre-filter can be applied to certain pixels in a layer.
    [0022] Figure 9 illustrates an example of how a layer is constrained within a bounding region.
    [0023] Figure 10 illustrates another example of how a layer is constrained within a bounding region.
    [0024] Figure 11 illustrates the use of a buffer region that is used during LSR processing.
    [0025] Figure 12 illustrates how the amount of LSR processing can be reduced in various situations.
    [0026] Figure 13 illustrates an exemplary method for improving LSR processing.
    [0027] Figure 14 illustrates another exemplary method for improving LSR processing.
    DETAILED DESCRIPTION OF THE INVENTION [0028] By way of introduction, the modalities described here refer to systems, hardware storage devices, and methods that improve LSR processing. To this end, the modalities described here generate a computer-generated scene that includes several layers. This scene is generated based on a predicted first pose from a device (for example, perhaps an HMD) from a computer system. A particular sub-region within one or more layers is identified and distinguished from at least one other region in one or more layers. This sub-region is then isolated
    Petition 870190079091, of 08/15/2019, p. 26/76
    7/40 of the other region (s). Subsequently, the particular LSR processing is applied to the isolated subregion while refraining from applying this particular LSR to another region (s).
    [0029] As will be discussed later, the modalities described here overcome many deficiencies prevalent throughout conventional technology. In addition, these modalities provide many other substantial benefits that will also be described in the remainder of this description.
    [0030] The following passage is provided as an initial guide to the subject of this description. In particle, this description will initially use Figure 1 to present some introductory discussion of a computing system. Following this discussion, some introductory subject on LSR processing is provided. So, this description exposes in Figure 2, which presents the subject in a computer system capable of improving the LSR processing performed in a multilayered scene. Figures 3-12 will then be discussed. These Figures provide details on using multiple layers in a computer generated scene where LSR is performed. Finally, additional supporting architectures and methods using the new principles described here will be detailed with respect to the subsequent Figures.
    [0031] As illustrated in Figure 1, in its most basic configuration, a computer system 100 takes many different forms. For example, Figure 1 illustrates computer system 100 as a desktop computer (or laptop) 100A, a tablet 1B, a cell phone 100C, or a head-mounted device 100D that includes an HMD. Although Figure 1 illustrates a wide range of computer devices, the current modalities are not limited only to those shown in Figure 1. As a result, ellipses 100E demonstrate that any type of compuPetition device 870190079091, of 15/08/2019 , p. 27/76
    8/40 tation is available for use by the present modalities.
    [0032] Computer system 100 typically includes at least one processing unit 110 and storage 120. Storage 120 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term memory can also be used here to refer to non-volatile mass storage as a physical storage medium. If the computing system is distributed, processing, memory, and / or storage capacity can be distributed as well. As used here, the term executable module, executable component, or even component can refer to software objects, routines, or methods that can be performed on the computer system. The different components, modules, engines, and services described here can be implemented as objects or processors that run on the computing system (for example, as separate topics).
    [0033] Modalities of the present invention may comprise or use a special purpose or general purpose computer including computer hardware, such as, for example, one or more processors (such as, processor 110) and system memory (such as, storage 120), as discussed in more detail below. Modes within the scope of the present invention also include physical or other computer-readable means for loading or storing executable instructions by computer and / or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are physical computer storage media. Computer-readable media that carry executable instructions
    Petition 870190079091, of 08/15/2019, p. 28/76
    9/40 per computer are means of transmission. Thus by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different types of computer-readable media: computer storage media and transmission media.
    [0034] Computer storage media comprise hardware, such as RAM, ROM, EEPROM, CD-ROM, solid state drives (SSDs) that are based on RAM, Flash memory, phase shift memory (PCM), or others types of memory, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other hardware medium that can be used to store desired program code media in the form of computer executable instructions, data or data structures, and that can be accessed by a general purpose or special purpose computer.
    [0035] As also shown in Figure 1, computer system 100 comprises several different components including I / O interface (s) 130, graphics rendering engine (s) 140, and one or more sensors 150. The computer system 100 can also be connected (via a wired or wireless connection) to external sensors 160 (for example, one or more remote cameras, accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). In addition, computer system 100 can also be connected via one or more wired or wireless networks 170 to remote systems 180 which are configured to perform any processing described with respect to computer system 100.
    [0036] During use, a user of computer system 100 is able to perceive information (for example, a computer generated scene) through a display screen that is included within the I / O interface (s) 130. The I / O interface (s) 130 and sensors 150/160 also
    Petition 870190079091, of 08/15/2019, p. 29/76
    10/40 include gesture detection devices, eye trackers, and / or other motion detection components (for example, cameras, gyroscopes, accelerometers, magnetometers, acoustic sensors, global positioning systems (GPS), etc.) that are capable of detecting the positioning and movement of one or more real-world objects, such as the user's hand, a pen, and / or any other object (s) that the user can interact with while immersed in the scene (ie, a mixed reality environment).
    [0037] In some cases, the positioning and movement of the user and objects are continuously monitored. This monitoring specifically detects any variation in the position and movement of objects, such as a detected change in position, speed, orientation or acceleration. These movements can be absolute movements and / or relative movements, as compared with a relative positioning of the HMD, and such that the movements / positioning of the HMD will be calculated in relative movements / positioning of the objects when they are presented in the scene.
    [0038] Graphics rendering engine 140 is configured, with processor (s) 110, to render one or more virtual objects within the computer generated scene. This rendering includes hand obstructions, or other types of obstructions, that are mapped to the relative positions of real-world objects. As a result, virtual objects move precisely in response to the movement of real-world objects. The graphics rendering engine 140 is also configured to render one or more obstructions that are purely virtual (without being mapped to real world objects), but which are, however, responsively positioned and moved for user input when the user interacts within the scene.
    Petition 870190079091, of 08/15/2019, p. 30/76
    11/40 [0039] The graphics rendering engine (s) 140, which can include one or more GPUs, are configured to render obstructions (ie, virtual objects) with certain display properties. These properties include coloring, transparency or opacity, texture, border definition (for example, thickness and / or sharpness vs. blur and / or diffusion), size, and so on. When certain movement limits are detected for the obstruction, then one or more combinations of the display properties for the obstruction will be modified (at least as long as the detected movement meets or exceeds the movement limits).
    [0040] Motion limits may include any combination of the following: (1) actual movements detected from the real world object associated with the obstruction, (2) animation movement detected from the virtual obstruction, and / or (3) detected variance or delay between real-world movement and animated movement. The detected movements are associated with speed and / or acceleration attribute values. These values are detected, measured and / or calculated by sensors 150 and / or processors 110 of computer system 100 to determine when predetermined limits have been met and / or exceeded.
    [0041] In some modalities, the same absolute movement limits are associated with all types of movements. For example, a certain speed, acceleration or delay associated with an obstruction will be sufficient to trigger a change in the obstruction's display properties. This change can be triggered regardless of the specific type of movement detected. In other embodiments, different types of movements (i.e., different movements within six degrees of freedom, including oscillation, elevation, agitation, inclination, rotation and yaw) are associated with different limits of movement. For example, a rotational acceleration
    Petition 870190079091, of 08/15/2019, p. 31/76
    12/40 a particular threshold will trigger a first limit that is different than a lateral acceleration.
    [0042] In some cases, there are at least two or more separate limits for one or more of the different types of movement. For example, in a first acceleration metric, the first limit is satisfied. In a second acceleration metric, which is greater or less than the first acceleration metric, a second limit is satisfied. In some cases, one or more separate speed limits are determined for each of the one or more types of movement. The appearance of the obstruction will dynamically change in response and each of the different limits being satisfied, such as changing the display attributes of the obstruction.
    [0043] The various limits and types of movement attributes are mapped into data structures stored in storage 120 of computing system 100 and / or in one or more remote systems 180.
    [0044] Some described modalities include GPU (s) 140 communicating with storage to determine when and how to gradually modify the appearance of the obstruction correspondingly to different detected changes in the movement attribute (s) that are associated with the obstruction . In such modalities, the display attributes / properties are continuously changed / scaled according to a continuously changing movement attribute (so that there are no discrete levels). For example, the levels of transparent and / or edge edges for an obstruction can be associated with many different corresponding magnitudes and the speed and / or acceleration associated with the obstruction (even when understanding real-world object movement) and / or associated delay with rendering an animation of the obstruction.
    Petition 870190079091, of 08/15/2019, p. 32/76
    13/40 [0045] A network such as network 170 shown in Figure 1, is defined as one or more data links and / or data switches that allow the transport of electronic data between computer systems, modules and / or others electronic devices. When information is transferred, or provided, over a network (either wired, wireless, or a combination of wired and wireless) on a computer, the computer appropriately views the connection as a means of transmission. Computer system 100 will include one or more communication channels that are used to communicate with network 170. The transmission means includes a network that can be used to load data or means of desired program code in the form of instructions executable by computer or in the form of data structures. In addition, these computer executable instructions can be accessed by a general purpose or special purpose computer. Combinations of the above must also be included within the scope of computer-readable media.
    [0046] Upon reaching various computer system components, the program code medium in the form of instructions executable by computer or data structures can be transferred automatically from the transmission media to computer storage media (or vice versa). For example, computer-readable instructions or data structures on a network or data link can be buffered in RAM inside a network interface module (for example, a network interface card or NIC) and then eventually transferred for computer system RAM and / or for less volatile computer storage medium on a computer system. Thus it is to be understood that computer storage media can be included in computer system components that also (or even primarily) use transmission media.
    Petition 870190079091, of 08/15/2019, p. 33/76
    14/40 [0047] Instructions executable by computer (or interpretable by computer) comprise, for example, instructions that make a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions . Computer executable instructions can be, for example, binary, intermediate format instructions such as assembly line, or meso source code. Although the subject has been described in specific language for structural features and / or methodological acts, it should be understood that the subject defined in the attached claims is not necessarily limited to the features described or acts described above. Instead, the characteristics and acts described as exemplary ways of implementing the claims.
    [0048] Those skilled in the art will appreciate that the invention can be practiced in networked computer environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, portable devices, communication systems. multiple processors, consumer programmable or microprocessor based electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches and the like. The invention can also be practiced in distributed system environments where the local and remote computer systems that are connected (either by wired data links, wireless data links, or by a combination of wired and wireless data links ) over a network, each performs tasks (for example, cloud computing, cloud services and the like). In a distributed system environment, program modules can be located on both local and remote memory storage devices.
    Petition 870190079091, of 08/15/2019, p. 34/76
    15/40
    Final Stage Reproiection (LSR) [0049] An issue in relation to generating a reality environment that is realistic is related to latency, or amount of time, in which images of virtual objects blocked in the world corresponding to a particular pose of a HMD are displayed to an HMD end user. For example, if a long time elapses between the time that the end user's head turns away from the particular pose and the time that an image of a virtual object is displayed based on the next pose, then the virtual object will appear to move away from its intended location within the mixed reality environment (that is, the image may not appear to be aligned with an intended real-world location or object). Consequently, HMDs can also include technology to generate and display some virtual objects at a frame rate that is greater than a rendering frame rate for other virtual objects.
    [0050] Virtual rendering objects at an increased frame rate significantly improve the stability of a scene. Displayed objects (ie, virtual objects) may include graphical end-stage adjustments to pre-rendered scenes (ie, forward predicted scenes that are rendered at a certain rendering frame rate) in order to incorporate frequency pose estimates bigger. The rendering frame rate can correspond with the minimum time to render images associated with an HMD pose. To achieve this goal, the HMD can do one or more of the following: 1) determine a predicted pose associated with a future position and orientation of the HMD (for example, a predicted 10 ms or 20 ms HMD pose in the future), generate a rendered image based on that predicted pose, 3) determine an updated pose associated with the subsequent HMD to generate the pre-rendered image or concurrent with the pre-rendered image being
    Petition 870190079091, of 08/15/2019, p. 35/76
    16/40 generated, 4) generate an updated image in the HMD. The updated image can be generated by means of a homographic transformation and / or a pixel offset adjustment of the pre-rendered image. In some cases, the updated image can be generated by circuit within the viewfinder. Additional details on these features will be provided later in this description.
    [0051] The predicted pose can be determined based on 1) a current position and orientation of the HMD and 2) an acceleration and a speed of the HMD just before determining the predicted pose (for example, extrapolating the predicted pose based on the movement of the HMD 5 ms, 10 ms or other predetermined distance before determining the predicted pose). The updated pose can be determined based on updated pose information that is provided to a pose tracker at a frequency greater than the rendering frame rate (for example, by obtaining tracking instrumentation data from various poses, such as accelerometers, gyroscopes, etc.).
    [0052] This updated pose information can be generated using a low latency inertial measurement unit (IMU) or combination of IMU and camera-based pose tracking. The updated image may comprise image rotation, translation, dilation (for example, stretching or shrinking), displacement, or skew of at least part of the pre-rendered image in order to correct the differences between the predicted pose and the updated pose (for example, to compensate for an incorrect pose prediction when generating the pre-rendered image).
    [0053] Furthermore, the updated image can be generated through a homographic transformation of the pre-rendered image. In some cases, the homographic transformation may include an affine transformation. The updated image can be generated using a pixel offset adjustment or a combination of transforms
    Petition 870190079091, of 08/15/2019, p. 36/76
    17/40 homographic sections and pixel offset adjustments. In some cases, homographic transformations and / or pixel offset adjustments can be generated downstream from the core rendering conduit (for example, using a controller or processor integrated with the HMD). In other cases, pixel displacement adjustments can be made using a display that incorporates displacement records or another circuit to allow displacement of pixel values within a pixel set of the display (for example, similar to the operation of a device coupled load).
    [0054] The updated images include final stage graphic adjustments of forwarding expected rendered images and can be generated using various image reprojection techniques of varying computational complexity. Image reprojection techniques can include pixel reprojection (for example, where each pixel in a rendered image is redesigned based on an updated pose), multi-layer homography (for example, where multiple rendered images associated with multiple layers within a 3D scene are used to generate the updated composite image), single layer homography (for example, where a single rendered image associated with a single layer within a 3D scene is used to generate the updated image), related homography, or adjustments based on pixel displacement.
    [0055] Although previous systems support LSR tuning, these systems fail to improve this processing in an efficient manner. As a result, these systems suffer many disadvantages associated with battery longevity. The described modalities can be used to help solve some of these problems by making selective LSR adjustments for multilayered scenes.
    Multilayered Computer Generated Scenes [0056] Attention will now be directed to Figures 2-12.
    Petition 870190079091, of 08/15/2019, p. 37/76
    18/40
    Briefly, Figure 2 illustrates an exemplary computer system that includes several components to enhance LSR processing in computer generated scenes that include multiple layers. Figures 3-12 expose in several functionalities that are available in the present modalities.
    [0057] As indicated above, Figure 2 illustrates a computer system 200 that includes several components (for example, a scene generation component 210, an LSR 220 component, a layer-creating component 230, and a rendering component. 240). Computer system 200 is analogous to computer system 100 shown in Figure 1. In some embodiments, computer system 200 is configured as a specialized set of one or more processors (for example, GPUs). Additional attributes of computer system 200 will also be discussed in connection with the remaining Figures.
    [0058] In particular, computer system 200 is capable of creating and rendering a computer generated scene. This scene can be a mixed, augmented or virtual reality scene. As previously discussed, this scene is generated based on information that was previously obtained by one or more hardware components of the computer system (for example, IMU instrumentation, accelerometers, gyroscopes, magnetometers, depth sensors, camera, infrared sensors, etc. .).
    [0059] Some modalities generate this scene in response to a predicted first pose of a part (for example, perhaps the HMD) of the computer system. This first predicted pose is different from an actual determined position previously obtained from the computer system. In an effort to reduce how often data is obtained from hardware components (whose process harms the battery life of the computer system),
    Petition 870190079091, of 08/15/2019, p. 38/76
    The present invention augments the actual pose data with predicted, or computer estimated, pose data. By performing this augmentation process, how often (that is, the frequency) the pose data obtained from the hardware components can be decreased. This increase process significantly reduces energy consumption. For clarity, assume that the computer system collects pose data from the various hardware components (for example, the IMU instrumentation discussed earlier) at a certain frequency. The present modalities reduce the frequency that these actual / current data are collected by increasing the actual pose data with predicted pose data. Consequently, as discussed throughout this description, the present modalities are capable of predicting the pose of a computer device based on 1) actual pose determinations obtained previously and 2) estimates regarding current and / or future behavior of the user ( for example, tracking body movement or other features).
    [0060] Continuing the discussion, the scene generation component 210 is capable of creating a scene that includes several input frames, such as the input frame 310 shown in Figure 3. The use of the term input frame correlates with the Scene term, meaning, the scene is actually comprised of several input frames that are displayed to a user using an HMD. As further shown in Figure 3, the input frame 310 includes a foreground layer (layer FG) and a bottom layer (layer BG). The FG 320 layer and the BG 330 layer are created by the layer creation component 230. In some situations, the FG 320 layer is rendered according to a first resolution, while the BG 330 layer is rendered according to a second resolution. For example, the FG 320 layer can include an object (for example, a tree) that appears to be visually closer to a
    Petition 870190079091, of 08/15/2019, p. 39/76
    20/40 user while the BG 330 layer can include a landscape (for example, mountains in the distant background) that appears to be further away from the user. Because the BG 330 layer includes content that appears to be virtually removed from the user, the content on the BG 330 layer can be rendered at a lower resolution.
    [0061] In other situations, computer system 100 supports foveated rendering. In view of this foveated rendering, the FG 320 layer can be placed in the higher resolution area (ie, the fovea area) while the BG 330 layer can be placed in a much lower resolution area (for example, an edge region or peripheral to the scene).
    [0062] The LSR 220 component is capable of performing LSR processing in the FG 320 layer and the BG 330 layer. In some cases, the LSR processing that is applied in the FG 320 layer is different from the LSR processing that is applied in the BG 330 layer. This difference in LSR processing between the FG 320 layer and the BG 330 layer can be due to a wide variety of reasons. By way of example and not limitation, the difference may be due to a determined depth difference between the FG 320 layer and the BG 330 layer. Movement of the HMD can also affect the positioning / rendering of the FG component differently from the positioning / rendering of the BG components .
    [0063] At this point, an example will be useful. Using the example above, it is assumed that the scene being presented to the user includes a tree that is nearby and a distant mountain landscape. Now, it is assumed that the user moves his position or otherwise changes his orientation. In response to the change in the user's position, the mixed reality scene will need to be updated. Here, it is worth noting that the change in visual effect for the tree, which is much closer to the user, will be significantly greater than what
    Petition 870190079091, of 08/15/2019, p. 40/76
    21/40 wants to move to the distant mountain landscape due to the perceived difference in depth. As a result, the LSR processing for each layer can be different.
    [0064] After LSR processing is applied to the various layers of the scene (for example, the FG 320 layer and the BG 330 layer), the computer system composes the layers together to form a unified layer, such as the unified layer 340 in Figure 3. In other words, the FG 320 layer and the BG 330 layer (after undergoing LSR processing) are composed together to form the unified layer 340. To ensure that the layers are properly visualized in the scene (for example, to ensure that layers are viewed with the correct depth and orientation in the scene (for example, to ensure that the layers are viewed with the correct depth and orientation with respect to each other), computer system 200 also applies one or more transformations to one or more These transformations can be applied either during LSR processing or during composite processing. For example, a layer may need to be resized (ie expanded), rotated, tilted or otherwise manipulated, which may not be required for the other layer (s) in the unified scene. To perform this selective manipulation, computer system 200 applies the appropriate type of transformation to allow the layer to be properly rendered. Some of these transformations include a homographic transformation, an inverted image transformation, an affine transformation, a pixel shift adjustment, depth processing transformations, and / or a matrix transformation. After composing the layers together, the rendering component 240 renders the unified layer 340 in an HMD.
    [0065] While Figure 3 illustrated an abstract visualization of
    Petition 870190079091, of 08/15/2019, p. 41/76
    22/40 a multilayered scene, Figures 4-7 illustrate a specific implementation.
    [0066] In particular, Figure 4 illustrates a first layer 400 of a computer generated scene. This first layer is analogous to the BG layer in Figure 3. This first layer 400 is illustrated as being bounded by a bounding region 410. This bounding region 410 refers to an area that is visible to a user using an HMD. Notably, however, the computer-generated scene is not restricted to just a single point of view. Instead, when the user moves the positions or otherwise changes the orientation, the view that is displayed on the HMD will change from the orientation of the HMD (for example, new content will appear while other content will disappear as the user moves. ). As a result, the first layer 400 and the boundary region 410 should not be used to characterize or otherwise limit the capabilities of the computer system 200.
    [0067] Similar to Figure 4, Figure 5 illustrates a second layer 500 that can be viewed by computer system 200. This second layer 500 is analogous to layer FG 320 in Figure 3. The second layer 500 is also illustrated as having a delimited region 510. Here, it is worth noting that the second layer 500 is slightly different from the first layer 400 of Figure 4. In particular, the second layer 500 includes one or more transparent parts 520. By transparent, it is understood that any content positioned below this transparent part 520 will be visible although below the content it is not a part of the second layer 500. As a result, after 1) the LSR processing of the second layer 500, 2) any transformations, and / or 3) the processes of composition are applied to the second layer 500 (and if layer 500 is still arranged on another layer in order to overlay this other layer),
    Petition 870190079091, of 08/15/2019, p. 42/76
    23/40 at least some parts of the lower layer will be visible through the transparent parts 520 of the second layer 500.
    [0068] Figure 6 illustrates a multilayered scene 600 presented as a composite, unified layer. This multilayered scene 600 is created by scene generation component 210. Multilayered scene 600 includes a first layer 610, which is analogous to the first layer 400 in Figure 4, and a second layer 620, which is analogous to the second layer 500 in Figure 5. In other words, this unified layer 600 is a composite of multiple layers. As further illustrated in Figure 6, the second layer 620 includes one or more transparent parts 630, which are analogous to the transparent parts 520 in Figure 5. Here, it is worth noting that a part of the first layer 610 is visible through the transparent part 630 of the second layer 620. To clarify, the second layer 620 obstructs, or obfuscates, several parts of the first layer 610. However, due to the fact that the second layer 620 includes the transparent parts 630, certain parts of the first layer 610 are visible through of the transparent parts 630 of the second layer 620 (at least while the user is in this particular pose). If the user later changed the position or orientation, then other parts of the first layer 610 will become visible through the transparent parts 630 of the second layer 620. The visible parts of which are directly related to the user's pose.
    [0069] Figure 7 is similar to Figure 6, but includes an additional layer, layer 710. In particular, Figure 7 illustrates a multilayered scene 700 which is similar to the multilayered scene 600 in Figure 6. Now, however, the multilayered scene 700 includes an additional layer, layer 710 (i.e., the tree). Layer 710 is positioned to overlap the other layers in the multilayer scene 700. Notably, a portion of layer 710 obstructs some
    Petition 870190079091, of 08/15/2019, p. 43/76
    24/40 but from the transparent parts of a lower layer (ie the house). As a result, layer 710 obstructs various parts of the other layers (for example, the tree covers the window, the window covers the mountain, but the mountain is visible through the window because the window is transparent, but the tree also covers a part of the Mountain).
    [0070] As discussed above, the various layers can undergo the same or different LSR processing, depending on the LSR processing being performed and movements detected from the HMD. In a worst case scenario, all pixels in all layers undergo LSR processing. Using Figure 3 as an example, the worst case scenario would occur if all pixels in layer BG 330 and all pixels in layer FG 320 were processed through LSR.
    [0071] In an effort to improve the efficiency of LSR processing, some of the modalities present pre-filter at least some of the pixels in the various layers of a multilayered scene. One method of pre-filtering pixels is to use a pixel mask. A pixel mask can be defined by row and column at the top, bottom, left and right edges of a particular layer. As mentioned in the discussion above, this pixel mask can be used to eliminate pixel processing, some of which may not contribute significantly to image quality (for example, pixels that are located in a border region of the display). This pre-filtering process allows some of the LSR processing to be ignored for individual pixels and even for large sections of pixels. This pre-filtering process is usually seen in Figure 8.
    [0072] In particular, Figure 8 shows that a pixel mask can be applied in one (or more) layer (for example, the BG 330 layer in Figure 3). Applying the pixel mask, LSR processing does not need to be performed, instead it can be omitted
    Petition 870190079091, of 08/15/2019, p. 44/76
    25/40 do / ignored, on pixels that have been identified by the pixel mask, operated by the pixel mask, or otherwise corresponding to the pixel mask.
    [0073] While Figure 8 described the use of a pixel mask, other types of operations can also be used to pre-filter the pixels. By way of example and not limitation, a vector mask can also be used to pre-filter some of the pixels (for example, identifying pixels to omit from LSR).
    [0074] Other modalities reduce or eliminate LSR by computing at each pixel location by applying a bounding box around certain layers. This bounding box functionality is shown in Figure 9. In particular, this bounding box is used to define an area that completely covers a certain layer (or a sub-region within a layer) without an output image (that is, a rendered scene). When using this bounding box, the various modalities eliminate all calculations outside the box when the output image changes (for example, when the user changes position or orientation and the scene adjusts accordingly). As illustrated in this passage, a bounding box is used to identify and isolate a sub-region within a particular layer. After this sub-region is isolated, then computer system 200 applies LSR processing to this sub-region while the preceding one applies LSR processing to the remaining regions of this layer. As a result, computer system 200 selectively determines which areas of a layer should have LSR processing applied to it.
    [0075] As discussed earlier, when a user changes position or orientation, the layers will need to be adjusted accordingly to reflect this change in position or orientation. For example, if the user moves to the right, the various pre layers
    Petition 870190079091, of 08/15/2019, p. 45/76
    26/40 will need to be adjusted to reflect the new point of view of the user. Here an example will be insightful. Suppose the user is viewing the multilayered scene 700 in Figure 7. When the user moves to the right, the user's view of the tree, house and mountain landscape will change. In particular, new areas of the tree and home are likely to be visible. The user's view of the tree and house in relation to the mountain landscape will also change. In contrast, the view of the mountainous landscape, which is too far away from the user, is unlikely to change, and if it does, the change is likely to be extremely small so as to be imperceptible. Because the tree is closer to the user than the house, more visual changes will be noticeable in relation to the change of perspective for the tree than those that will be noticeable in relation to the house. Consequently, the computer system supports the application of different LSR processing at different layers and which can be triggered by detecting certain movement limits for different layers. Independent triggers can be applied to separate threshold determinations for each layer.
    [0076] Continuing with the discussion, to carry out this adjustment in response to the user's change in position, computer system 200 selectively applies various transformations to the layers. These transformations are applied during LSR processing and / or during a final composite process. Such a transformation, as briefly mentioned earlier, is an inverted image transformation. This type of transformation determines the corners of a layer (for example, perhaps using the bounding box in Figure 9) using image resolution techniques. This transformation is then inverted to determine the point on the unified layer (that is, the output image) that corresponds to the upper left corner of the input image, as is currently shown in Figure 10.
    Petition 870190079091, of 08/15/2019, p. 46/76
    27/40
    Similar calculations are performed to determine the other corners. Once the layer is identified, then the layer can be manipulated in a variety of ways (for example, stretched, shrunk, tilted, etc.). As discussed earlier, other types of transformations include homographic transformations, affine transformations, pixel shift adjustments, depth processing transformations, and matrix transformations.
    [0077] Here, it is worth noting that not all transformations are easily reversed. For example, large-scale movements can be performed with a matrix transformation. However, minor adjustments can be made using other transformations (for example, depth processing transformations). To account for these transformations, some modalities impose a buffer region around the previously generated bounding box. This buffer region is illustrated in Figure 11 and can be used to help blend the various layers together to make a unified layer visually attractive and cohesive.
    [0078] In many cases, such as an HMD supports foveated rendering, a bottom layer will be completely obstructed by a foreground layer after a small amount of mixing has been done. The present modalities capitalize on situations such as these by reducing, or even eliminating entirely, the amount of processing that is applied to those obstructed parts. For example, Figure 12 shows the foreground layer that is opaque and that obstructs a part of the bottom BG layer. For those parts of the lower layer that are obstructed by the opaque regions of an upper layer, LSR calculations can be ignored and the battery consumption can be reduced.
    [0079] Having just discussed a computer system that improves LSR processing in a multilayered scene,
    Petition 870190079091, of 08/15/2019, p. 47/76
    28/40 the description will now present several exemplary methods for improving LSR processing. Notably, the following discussion refers to a number of methods and method acts that can be performed. Although the acts of the method may be discussed in a certain order or illustrated in a flow chart as occurs in a particular order, no particular order is required of them that is specifically established, or required because an act is dependent on another act being completed before the act being performed.
    Selective Application of Reproiection Processing in Layer Sub-Regions [0080] Figure 13 illustrates an exemplary method 1300 that is performed by a computer system, such as computer system 200 in Figure 2. In particular, method 1300 includes an act (act 1310) of generating a scene that includes one or more layers.
    [0081] Here, it is worth noting that in some cases, an entire layer can be comprised of only a single element such that the element is the layer. Using Figure 5 as an example, the entire layer 500 is comprised of only a single housing element. In other cases, however, a layer can be comprised of multiple elements. By way of example and not limitation, Figure 4 illustrates that layer 400 includes a mountain landscape, a road, the sky, and perhaps a pasture. Here, the mountain landscape can be considered a different element, the road can be considered a different element, etc. Therefore, in some cases, layer 400 is comprised of multiple distinct elements. In other modalities, however, there is no such distinction between elements in a layer.
    [0082] In some modalities, the scene is generated based on several poses of a part (for example, the HMD) of a computer system. To clarify, the poses include certain actual poses
    Petition 870190079091, of 08/15/2019, p. 48/76
    29/40 swims by IMU instrumentation as well as predicted poses determined by estimated behavior and anticipated movement. Here, the scene generation component 210 generates the scene in the manner previously described. Additional description of this feature will be presented later in this description.
    [0083] Some modalities configure this scene for foveated rendering. In particular, these modalities use eye tracking technology to track the user's eye movements. These modalities then render the scene so that at least one of the layers is displayed as being close to a fovea in the user's eye while other layers are not displayed as being close to the fovea of the user's eye. Correspondingly, if modalities distinguish between elements within a layer, then the elements are also displayed according to foveated rendering techniques.
    [0084] Method 1300 also includes an act (act 1320) of identifying a sub-region within one or more layers. This sub-region is distinguished from at least one other region in one or more layers. Here, the layer-creating component 230 manipulates and otherwise distinguishes sub-regions within the various layers.
    [0085] In some embodiments, the process of identifying this sub-region also includes identifying 1) a particular element within one or more layers and 2) a buffer limit around the particular element. Therefore, in these modalities, the sub-region includes the element itself as well as the buffer limit around the element. In contrast, other modalities do not include such a buffer limit around the sub-region.
    [0086] In the example above, layer 400 in Figure 4 is comprised of multiple elements (for example, the mountain landscape element, the road element, etc.). Some selections
    Petition 870190079091, of 08/15/2019, p. 49/76
    30/40 provide an entire element (for example, the road element) to serve as the sub-region. Alternatively, however, the identified sub-region can only be a generalized layer part with no particular relation to an element.
    [0087] In some embodiments, the particular element is identified in response to detecting a particular movement limit that has been dyed or exceeded for the element with respect to the relative HMD, which has not been achieved for the relative movement of other elements / layers with respect to to the HMD.
    [0088] Method 1300 then includes an act (act 1330) of isolating the identified sub-region from at least one other region of one or more layers. Similar to the step above, the layer creation component 230 also performs this act. This isolation is accomplished in a variety of ways. For example, this isolation is accomplished by applying a pixel mask or a vector mask to one or more layers. As detailed above, a buffer region can also be applied around this isolated sub-region. This buffer region is created in the manner previously described in this invention.
    [0089] Continuing with the example above, assuming that the road element is selected as the sub-region. This road element is then distinguished from the other regions in layer 400 (for example, the other regions at least include the mountainous landscape element). To distinguish the road element, the modalities isolate the road element from other regions using a pixel mask, vector mask, bounding box, etc.
    [0090] Finally, Figure 13 shows an act (act 1340) of applying final stage reprojection processing in the isolated sub-region while refraining from applying final stage reprojection processing in at least another region of one or more layers . Per
    Petition 870190079091, of 08/15/2019, p. 50/76
    31/40 example means, the LSR component 220 performs LSR processing in the manner previously described. In some embodiments, LSR processing includes one or more transformations to transform the one or more layers or the elements within the one or more layers. These transformations include a homographic transformation, an affine transformation, an inverted image transformation, a pixel shift adjustment, a depth processing transformation, a matrix transformation, or any other type of image transformation (stretch, stretch, rotate , etc.). In addition, one or more transformations are applied to one or more layers (or the elements within those layers) in response to a predicted pose of a part (for example, the HMD) of the computer system. Here, it is worth noting that in some multilayered scenes, LSR processing is applied to all pixels in one layer while LSR processing is applied only to a subset of pixels in a different layer. In other situations, however, LSR processing is applied to all pixels on all layers. This situation is often the worst case scenario.
    [0091] Continuing with the example above, after the road element is selected for isolation, the modalities perform LSR processing on this road element. For example, suppose the user changes his position or orientation in some way (for example, the user may go from a standing position to a sitting position). The parts of the road more distant from the user probably will not have changed much in perspective or view. However, parts of the road closer to the user are likely to have changed significantly in perspective in response to the user's position change, thereby exceeding the movement limit in full distance / acceleration. In response to this new pose, these fashion
    Petition 870190079091, of 08/15/2019, p. 51/76
    32/40 lities perform LSR processing on the identified element (for example, the road or a particular part of the road).
    [0092] As indicated above, this LSR processing can also include the selective application of various transformations. Here, certain transformations will need to be applied to the road element to ensure that its visualization is properly rendered in response to the user's position change. For example, the part of the road closest to the user will likely have to be stretched, tilted or otherwise manipulated to ensure that the road is properly aligned and perceived by the user. As a result, the LSR processing that is applied to the identified sub-region (for example, the road element at layer 400) may differ from the LSR processing that is applied to the remaining regions within this layer (for example, perhaps no LSR processing was applied to the mountainous landscape element because this element is perceived to be very far from the user and will change very little in response to changes in the user's position).
    [0093] In scenes that include multiple layers, a part of a layer (for example, the top layer) will overlap a part of a bottom layer. As a result, the part of the upper layer obscures an obscured part of the lower layer. In such multi-layered scenes, the identified sub-region may be included within the part of the upper layer that obscures the obfuscated part of the layer below. Here, it is worth noting that the sub-region identified, in some cases, will include a transparent part (for example, the transparent part 630 of Figure 6). In these situations, the part of the layer below that is located below the transparent part of the sub-region will be visible through the transparent part, even after LSR processing is applied.
    [0094] Consequently, the modalities support the application
    Petition 870190079091, of 08/15/2019, p. 52/76
    33/40 selective LSR processing in layer sub-regions of a multilayered scene.
    Selectively Apply Replay Processing in Multilayer Scenes [0095] Figure 14 illustrates another exemplary method 1400 that is performed by a computer system, such as computer system 200 in Figure 2, where each element of the scene can comprise a layer different.
    [0096] Initially, method 1400 includes an act (act 1410) of generating a scene that includes several layers. Here, scene generation component 210 and layer creation component 230 work together to generate scenes with multiple layers. As above, this scene can be included as part of the mixed, augmented or virtual reality scene.
    [0097] In some modalities, this scene is generated in response to the various poses of a part (for example, the HMD) of the computer system. These poses include 1) actual poses determined by IMU instrumentation and 2) predicted poses based on user behavior (eg, historical and present behavior) and anticipated movement. When a predicted pose is generated, it is based on previous pose determinations. Additionally, the predicted pose data is used to augment the actual pose data so that the computer system reduces the frequency with which it obtains data from its hardware IMU instrumentation. Reduce how often the data is actually collected, the modalities extend the battery life of the computer system.
    [0098] Multiple pose determinations can be generated. For example, some modalities generate multiple pose determinations predicted for a particular point in time (for example, ten different pose estimates for time X), where each determination of
    Petition 870190079091, of 08/15/2019, p. 53/76
    34/40 predicted pose is an estimate of how the HMD will be positioned at this point in time. These modalities assign a probability of accomplishment for each of those predicted pose determinations (for example, pose Y has a 52% probability of being performed, pose Z has a 35% probability of being performed, etc.). This probability is determined based on 1) historical user behavior, 2) current user movements, and / or 3) events that are currently occurring or that will occur right at the scene (for example, the software application determines that a new enemy will appear right on the scene at a particular location and time and the software application estimates that the user will react to that new appearance by making a certain movement). After the probability determinations are complete, the modalities select the highest predicted pose and continue with the following operations.
    [0099] Other modalities generate several determinations of predicted pose in a different way. Here, instead of generating several predicted pose determinations for a single point in time, these modalities are based on successive predicted pose determinations on previous predicted pose determinations. To clarify, suppose that the computer system collects actual pose data at a certain frequency, perhaps every 20 ms (this number is an example only and should not be considered as a limiting metric and can be more or less than 20 ms) . Notably, however, the scene may need to be updated at a faster rate. As a result, these modalities increase the actual pose data with predicted pose data. To do so, these modalities generate a first predicted pose based at least partially on the actual pose data and render the scene based on the first predicted pose. Then, the modalities generate a second predicted pose based at least partially on the first predicted pose and then
    Petition 870190079091, of 08/15/2019, p. 54/76
    35/40 render the scene based on this predicted second pose. This process continues, where a successive predicted pose is based on a previous predicted pose until the computer system again obtains actual pose data. As a result, successive predicted poses are different from an actual determined pose previously obtained from the computer system. In some situations (and in response to the various poses provided for), the LSR processing that is applied to the multiple layers in the scene is applied to all pixels of those layers. In other situations, however, LSR processing is applied only to a subset of pixels within one or more layers.
    [00100] Method 1400 also includes an act (act 1420) of applying LSR processing on a first layer in the various layers. Here, the LSR 220 component applies LSR processing to the layers. Notably, some modalities apply LSR processing only to an identified subset of pixels within a particular layer (for example, perhaps some of the pixels have been pre-filtered from the layer in the manner described earlier in this description) while other modalities apply LSR processing in all pixels of the multiple layers.
    [00101] Method 1400 also includes an act (act 1430) of applying LSR processing in a second layer in the various layers. The LSR processing applied to this second layer is carried out independently of the LSR processing of the first layer. In addition, the LSR processing applied to the second layer includes one or more transformations (for example, an inverted image transformation and / or a matrix transformation). These transformations are applied to the second layer, but are not applied to the first layer. The different transformations that are applied to the first layer and the second layer may differ, based on
    Petition 870190079091, of 08/15/2019, p. 55/76
    36/40 in detecting different attributes of different layers and / or based on detecting which different movement limits were reached or not for different layers. Stored and accessed parameters define when different transformations must be applied to different layers.
    [00102] Additional transformations can also be applied during a subsequent composite process that will be described later. Therefore, image transformations can be applied to layers at several different times. Similar to the above act, the LSR 220 component applies this LSR processing. As a result, the LSR 220 component is capable of applying different LSR processing to different layers at different times.
    [00103] Here, it is worth noting that a wide variety of transformations can be applied to layers. In addition to the aforementioned transformations (i.e., homographic transformations, related transformations, inverted image transformations, depth processing transformations, pixel shift transformations, matrix transformations, etc.), the transformations also include 1) a translation, 2) a reflection, 3) a rotation, or 4) an expansion transformation. Several other transformations can be applied to change other characteristics of a layer. By way of example, supposing in response to a change in the user's position, a new brightness or reflection must be introduced into the scene (for example, perhaps the computer system determines that sunlight should now reflect outside the house window in the Figure 5 and must be projected in the user's eyes). This new glow or reflection will now impact how the user views the scene, particularly the color and transparency aspects of the various layers. As a result, the color or transparency of the various layers in the scene may need to be changed. Therefore, in addition to size transformations, one or more
    Petition 870190079091, of 08/15/2019, p. 56/76
    37/40 more transformations also include color and transparency transformations.
    [00104] Method 1400 also includes an act (act 1440) of creating a unified layer. This unified layer is created by composing the various layers together (for example, the second layer transformed with the first layer). Here, the layer creation component 230 creates this unified layer.
    [00105] Some modalities carry out this composition by generating a buffer region around the second layer in the manner that was previously described in this description. Additionally, composing the transformed second layer with the first layer can make the transformed second layer overlap a part of the first layer. As a result, the transformed second layer obscures at least part of the first layer. However, some of the layers may include a transparent part (s). Due to this transparent part (s), the content that is below the transparent part (s) is visible through the transparent part (s).
    [00106] Finally, method 1400 includes an act (act 1450) of rendering the unified layer. This unified layer can be rendered in an HMD. Here, rendering component 240 renders the unified layer.
    [00107] Some modalities support foveated rendering in HMD so that the unified layer is rendered using foveated rendering techniques. To clarify, if the underlying computer system includes an HMD, then rendering the unified layer is done by rendering the unified layer in the HMD.
    [00108] Although not shown in Figure 14, other modalities include acts of pre-filtering pixels of several layers. For example, some pixels from the first layer can be pre-filtered from other pixels from the first layer. Alternatively, some pixels of the
    Petition 870190079091, of 08/15/2019, p. 57/76
    38/40 first layer can be pre-filtered from pixels of a different layer (for example, in the case of a composite, unified layer). This pre-filtering process is carried out by applying a pixel mask or a vector mask. In these modalities, at least some of the pixels that are pre-filtered are identified as being border pixels. These border pixels are determined to contribute less to scene quality than unfiltered or borderless pixels. Because these pixels contribute less to the quality of the scene, the modalities optimize the battery life by ignoring the LSR processing on these pixels.
    [00109] Other modalities generate additional predicted poses of the part (for example, the HMD) of the computer system. These modalities create a new unified layer composing the second layer again with the first layer. In some cases, the part of the first layer that was previously visible through a transparent part of the second layer is now overshadowed in response to changes in orientation associated with the predicted next pose. Now, a new part of the first layer that was not originally visible through the transparent part of the second layer becomes visible through the transparent part in response to changes in orientation associated with the predicted next pose (for example, a new view of the mountain landscape through the window shown in Figure 6 is now visible).
    [00110] Some modalities also determine when a previously overlapped part of a first layer is still overlapped by a second layer even in response to the next predicted pose of the computer system part (for example, the part of the mountainous landscape near the bottom of the house in Figure 6 is still superimposed). In response, these modalities ignore the processing of the overlapping part of the first layer. For school
    Petition 870190079091, of 08/15/2019, p. 58/76
    39/40 recer, the modalities apply LSR processing to some parts of the first layer while refraining from applying LSR processing to the superimposed part of the first layer that is still superimposed by the second layer. As a result, the modalities dynamically determine when LSR processing is preventable and efficiently optimize the battery life of the computer system by ignoring LSR processing during those avoidable cases.
    [00111] It will be appreciated that while most of this description emphasizes generating pose data based on a pose from an HMD, this will not always be the case. For example, the described modalities are also capable of generating relational pose data between an HMD and a device in the different body (for example, a hand device, a foot device, a chest device, etc.) or even a device that it is not positioned on the user's body (for example, a base station that is positioned in a room and that is used to further improve the mixed reality experience by mapping and detailing the contents of the room). Therefore, in addition to generating a scene based on the HMD pose, the modalities are also capable of generating the scene based on a relationship between the HMD and the various devices in the body and / or base. Here, a brief example will be useful. Suppose the user is using an HMD and is holding a portable device. When immersed in a scene, a laser gun can be viewed as being attached to the user's arm. To calculate the correct orientation and pose of the laser gun, the computer system relies on pose data collected from the HMD and the handheld device. Consequently, the modalities discussed here are not limited to generating a scene based only on pose data obtained from an HMD. Instead, modalities are also able to generate a scene based on pose data collected from other types of devices on the body and / or base.
    Petition 870190079091, of 08/15/2019, p. 59/76
    40/40 [00112] Consequently, this description introduced systems, hardware storage devices, and methods that optimize LSR processing for computer generated scenes that include multiple layers. Some modalities make LSR processing selectively applied to several sub-regions of the layer. Other modalities make LSR processing selectively applied to multilayered scenes. As a result of these processes, the modalities presented here optimize and selectively apply LSR processing and can therefore help to conserve energy that would otherwise be spent on universal LSR processing.
    [00113] The present invention can be incorporated in other specific forms without departing from its spirit and characteristics. The described modalities should be considered in all aspects only as illustrative and not restrictive. The scope of the invention, therefore, is indicated by the appended claims instead of the preceding description. All changes that fall within the meaning and equivalence range of the claims must be covered within its scope.
    权利要求:
    Claims (15)
    [1]
    1. Computer system, characterized by the fact that it comprises:
    one or more processors; and one or more computer-readable hardware storage media having stored in the same computer-executable instructions, the computer-executable instructions being executable by one or more processors to make the computer system optimize the final stage redesign by making the computer system :
    generate a scene that includes one or more layers, the scene being generated based on a predicted first pose of a part of the computer system;
    identify a sub-region within one or more layers, the sub-region being distinguished from at least one other region in the one or more layers;
    isolating the subregion from at least one other region from one or more layers; and applying final stage reprojection processing in the isolated sub-region while refraining from applying final stage reprojection processing in at least one other region of one or more layers.
    [2]
    2. Computer system according to claim 1, characterized by the fact that the final stage reprojection processing includes applying an inverted image transformation to one or more layers.
    [3]
    3. Computer system, according to claim 1, characterized by the fact that isolating the sub-region is accomplished at least partially by applying a pixel mask on one or more layers.
    Petition 870190079091, of 08/15/2019, p. 61/76
    2/4
    [4]
    4. Computer system according to claim 1, characterized by the fact that identifying the sub-region includes identifying (1) a particular element within one or more layers and (2) a boundary around the particular element, and where the sub-region includes the element and the boundary around the particular element.
    [5]
    5. Computer system, according to claim 1, characterized by the fact that the final stage reprojection processing comprises one or more transformations to transform the one or more layers, the one or more transformations being applied to the one or more layers in response to the first predicted pose of the computer system.
    [6]
    6. Computer system, according to claim 1, characterized by the fact that the execution of executable instructions by computer still causes the computer system to apply a buffer region around the isolated subregion.
    [7]
    7. Hardware storage devices, characterized by the fact that they have stored in the same computer executable instructions that are executable by one or more processors of a computer system to make the computer system optimize the final stage redesign by making the system computer:
    generate a scene that includes one or more layers, the scene being generated based on a predicted first pose from a part of the computer system;
    identify a sub-region within one or more layers, the sub-region being distinguished from at least one other region in the one or more layers;
    isolating the subregion from at least one other region from one or more layers; and apply final stage reprojection processing to
    Petition 870190079091, of 08/15/2019, p. 62/76
    3/4 isolated sub-region while refraining from applying final stage reprojection processing in at least one other region of one or more layers.
    [8]
    8. Hardware storage devices according to claim 7, characterized by the fact that isolating the subregion is at least partially accomplished by applying a vector mask on one or more layers.
    [9]
    Hardware storage devices according to claim 7, characterized by the fact that the final stage reprojection processing includes applying a matrix transformation in one or more layers.
    [10]
    10. Hardware storage devices according to claim 7, characterized by the fact that the first predicted pose is determined based on information previously obtained by one or more hardware instruments of the computer system.
    [11]
    11. Hardware storage devices according to claim 7, characterized by the fact that the one or more layers include (1) a first layer that is rendered according to a first resolution and (2) a second layer that is rendered according to a second resolution.
    [12]
    12. Hardware storage devices according to claim 11, characterized by the fact that the computer system also tracks the eye movements of a computer system user, and in which the scene elements included within the first layer are displayed near a user's eye fovea while the scene elements included within the second layer are not displayed near the user's eye fovea, where the scene is configured for foveated rendering.
    [13]
    13. Hardware storage devices, according
    Petition 870190079091, of 08/15/2019, p. 63/76
    4/4 of claim 7, characterized in that the one or more layers include a first layer and a second layer, and in which the final stage reprojection processing that is applied to the first layer is different from the reprojection processing final stage that is applied to the second layer.
    [14]
    14. Hardware storage devices according to claim 13, characterized by the fact that the difference between the final layer reprojection processing of the first layer and the final layer reprojection processing of the second layer is due to a difference in depth determined between the first layer and the second layer.
    [15]
    15. Method to optimize the re-projection of the final stage the method being performed by one or more processors of a computer system, characterized by the fact that the method comprises:
    generate a scene that includes one or more layers, the scene being generated based on a predicted first pose of a part of the computer system;
    identify a sub-region within one or more layers, the sub-region being distinguished from at least one other region in the one or more layers;
    isolating the subregion from at least one other region from one or more layers; and applying final stage reprojection processing in the isolated sub-region while refraining from applying final stage reprojection processing in at least one other region of one or more layers.
    类似技术:
    公开号 | 公开日 | 专利标题
    BR112019016952A2|2020-04-14|selective application and reprojection processing in layer sub-regions to optimize the final stage reprojection energy
    US10514753B2|2019-12-24|Selectively applying reprojection processing to multi-layer scenes for optimizing late stage reprojection power
    EP2732436B1|2019-09-18|Simulating three-dimensional features
    US9978180B2|2018-05-22|Frame projection for augmented reality environments
    US9978118B1|2018-05-22|No miss cache structure for real-time image transformations with data compression
    US11176737B2|2021-11-16|Textured mesh building
    JP2014525089A5|2015-09-10|
    US10600150B2|2020-03-24|Utilizing an inertial measurement device to adjust orientation of panorama digital images
    US20110261048A1|2011-10-27|Electronic device and method for displaying three dimensional image
    BR102017019134A2|2018-05-02|AUTOMATIC ZOOM METHOD, DEVICES AND SYSTEMS WHEN EXECUTING AN INCREASED REALITY SCENE
    US10242654B2|2019-03-26|No miss cache structure for real-time image transformations
    US10127886B2|2018-11-13|Modifying hand occlusion of holograms based on contextual information
    US8922591B2|2014-12-30|Holographic display
    AU2018245993B2|2022-03-03|Selective application of reprojection processing on layer sub-regions for optimizing late stage reprojection power
    US11023035B1|2021-06-01|Virtual pinboard interaction using a peripheral device in artificial reality environments
    CN106780659A|2017-05-31|A kind of two-dimension situation map generalization method and electronic equipment
    JP2020537200A|2020-12-17|Shadow generation for image content inserted into an image
    US20180290055A1|2018-10-11|Rhythm game displaying method and terminal for performing the method
    WO2019051228A1|2019-03-14|Camera-based transparent display
    同族专利:
    公开号 | 公开日
    WO2018183025A1|2018-10-04|
    IL268797D0|2019-10-31|
    CL2019002687A1|2020-02-14|
    ZA201905233B|2020-10-28|
    RU2762123C2|2021-12-15|
    IL268797A|2022-01-01|
    SG11201908293QA|2019-10-30|
    JP2020518885A|2020-06-25|
    CA3053408A1|2018-10-04|
    US20180276824A1|2018-09-27|
    US10410349B2|2019-09-10|
    PH12019550174A1|2020-06-08|
    AU2018245993A1|2019-08-22|
    CN110462555A|2019-11-15|
    MX2019011541A|2019-12-11|
    CO2019009849A2|2019-09-30|
    KR20190133716A|2019-12-03|
    EP3602248A1|2020-02-05|
    CN110462555B|2021-12-03|
    RU2019134184A3|2021-07-05|
    RU2019134184A|2021-04-28|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    US5864342A|1995-08-04|1999-01-26|Microsoft Corporation|Method and system for rendering graphical objects to image chunks|
    US6005582A|1995-08-04|1999-12-21|Microsoft Corporation|Method and system for texture mapping images with anisotropic filtering|
    US5933125A|1995-11-27|1999-08-03|Cae Electronics, Ltd.|Method and apparatus for reducing instability in the display of a virtual environment|
    US6057847A|1996-12-20|2000-05-02|Jenkins; Barry|System and method of image generation and encoding using primitive reprojection|
    US6111582A|1996-12-20|2000-08-29|Jenkins; Barry L.|System and method of image generation and encoding using primitive reprojection|
    JP3356663B2|1997-10-22|2002-12-16|松下電器産業株式会社|Image encoding device, image encoding method, and recording medium recording image encoding program|
    US6456340B1|1998-08-12|2002-09-24|Pixonics, Llc|Apparatus and method for performing image transforms in a digital display system|
    US6369830B1|1999-05-10|2002-04-09|Apple Computer, Inc.|Rendering translucent layers in a display system|
    US7071935B1|1999-06-14|2006-07-04|Sun Microsystems, Inc.|Graphics system with just-in-time decompression of compressed graphics data|
    US6901169B2|2001-02-01|2005-05-31|At & T Corp.|Method and system for classifying image elements|
    US9113130B2|2012-02-06|2015-08-18|Legend3D, Inc.|Multi-stage production pipeline system|
    US7324695B2|2002-03-18|2008-01-29|Seimens Medical Solutions Usa, Inc.|Prioritized image visualization from scalable compressed data|
    US7199793B2|2002-05-21|2007-04-03|Mok3, Inc.|Image-based modeling and photo editing|
    WO2005039185A1|2003-10-06|2005-04-28|Mindego, Inc.|System and method for creating and executing rich applications on multimedia terminals|
    US7369134B2|2003-12-29|2008-05-06|Anark Corporation|Methods and systems for multimedia memory management|
    JP4861338B2|2005-02-15|2012-01-25|コーニンクレッカフィリップスエレクトロニクスエヌヴィ|Improving the performance of the memory unit of the data processing device by separating the read function and the fetch function|
    US20070050605A1|2005-08-29|2007-03-01|Bran Ferren|Freeze-dried ghost pages|
    US20090016333A1|2006-06-14|2009-01-15|Derek Wang|Content-based adaptive jitter handling|
    US9117309B1|2005-12-19|2015-08-25|Nvidia Corporation|Method and system for rendering polygons with a bounding box in a graphics processor unit|
    US9019300B2|2006-08-04|2015-04-28|Apple Inc.|Framework for graphics animation and compositing operations|
    US20080158254A1|2006-12-29|2008-07-03|Hong Jiang|Using supplementary information of bounding boxes in multi-layer video composition|
    US7982733B2|2007-01-05|2011-07-19|Qualcomm Incorporated|Rendering 3D video images on a stereo-enabled display|
    JP4857196B2|2007-05-31|2012-01-18|キヤノン株式会社|Head-mounted display device and control method thereof|
    US9648325B2|2007-06-30|2017-05-09|Microsoft Technology Licensing, Llc|Video decoding implementations for a graphics processing unit|
    US9665951B2|2007-12-20|2017-05-30|Telefonaktiebolaget Lm Ericsson |Unified compression/decompression graphics architecture|
    US9355493B2|2007-12-31|2016-05-31|Advanced Micro Devices, Inc.|Device and method for compositing video planes|
    JP5120987B2|2008-07-24|2013-01-16|トムソンライセンシング|Apparatus, method and system for image processing|
    US8106924B2|2008-07-31|2012-01-31|Stmicroelectronics S.R.L.|Method and system for video rendering, computer program product therefor|
    CN102282840B|2009-01-19|2016-01-06|杜比实验室特许公司|Multiplexed imaging|
    US8707179B2|2009-09-25|2014-04-22|Avazap, Inc.|Frameless video system|
    US8416263B2|2010-03-08|2013-04-09|Empire Technology Development, Llc|Alignment of objects in augmented reality|
    US9077970B2|2011-02-25|2015-07-07|Adobe Systems Incorporated|Independent layered content for hardware-accelerated media playback|
    US9262950B2|2011-04-20|2016-02-16|Microsoft Technology Licensing, Llc|Augmented reality extrapolation techniques|
    CN103597839B|2011-05-31|2017-10-20|杜比实验室特许公司|Video-frequency compression method, video reconstruction method and system and encoder|
    US9602766B2|2011-06-24|2017-03-21|At&T Intellectual Property I, L.P.|Apparatus and method for presenting three dimensional objects with telepresence|
    AU2012348348B2|2011-10-28|2017-03-30|Magic Leap, Inc.|System and method for augmented and virtual reality|
    US9153062B2|2012-02-29|2015-10-06|Yale University|Systems and methods for sketching and imaging|
    US10019833B2|2012-03-20|2018-07-10|Siemens Corporation|Luggage visualization and virtual unpacking|
    US9098418B2|2012-03-20|2015-08-04|Apple Inc.|Coordinated prefetching based on training in hierarchically cached processors|
    US8687902B2|2012-03-29|2014-04-01|Intel Corporation|System, method, and computer program product for decompression of block compressed images|
    US9576397B2|2012-09-10|2017-02-21|Blackberry Limited|Reducing latency in an augmented-reality display|
    CN103873758B|2012-12-17|2018-09-21|北京三星通信技术研究有限公司|The method, apparatus and equipment that panorama sketch generates in real time|
    WO2014160342A1|2013-03-13|2014-10-02|The University Of North Carolina At Chapel Hill|Low latency stabilization for head-worn displays|
    CN113918018A|2013-03-15|2022-01-11|奇跃公司|Display system and method|
    US9443355B2|2013-06-28|2016-09-13|Microsoft Technology Licensing, Llc|Reprojection OLED display for augmented reality experiences|
    US9582075B2|2013-07-19|2017-02-28|Nvidia Corporation|Gaze-tracking eye illumination from display|
    US9514571B2|2013-07-25|2016-12-06|Microsoft Technology Licensing, Llc|Late stage reprojection|
    EP2833326B1|2013-07-30|2016-04-06|Dassault Systèmes|Lossless compression of a 3D mesh including transforming of the mesh to a image|
    US10726560B2|2014-10-31|2020-07-28|Fyusion, Inc.|Real-time mobile device capture and generation of art-styled AR/VR content|
    CN106105220B|2014-01-07|2019-07-05|诺基亚技术有限公司|For Video coding and decoded method and apparatus|
    US10935788B2|2014-01-24|2021-03-02|Nvidia Corporation|Hybrid virtual 3D rendering approach to stereovision|
    US9652893B2|2014-04-29|2017-05-16|Microsoft Technology Licensing, Llc|Stabilization plane determination based on gaze location|
    US9996976B2|2014-05-05|2018-06-12|Avigilon Fortress Corporation|System and method for real-time overlay of map features onto a video feed|
    US9430393B2|2014-05-26|2016-08-30|Texas Instruments Incorporated|System and method for managing cache|
    US20150378920A1|2014-06-30|2015-12-31|John G. Gierach|Graphics data pre-fetcher for last level caches|
    US20150379772A1|2014-06-30|2015-12-31|Samsung Display Co., Ltd.|Tracking accelerator for virtual and augmented reality displays|
    US9761001B2|2014-09-25|2017-09-12|Intel Corporation|Filtered shadow mapping|
    US9659410B2|2014-10-21|2017-05-23|Honeywell International Inc.|Low latency augmented reality display|
    US20160136816A1|2014-11-14|2016-05-19|James Charles Pistorino|Sorting apparatus and method|
    US9412169B2|2014-11-21|2016-08-09|iProov|Real-time visual feedback for user positioning with respect to a camera and a display|
    US9824498B2|2014-12-30|2017-11-21|Sony Interactive Entertainment Inc.|Scanning display system in head-mounted display for virtual reality|
    US9756375B2|2015-01-22|2017-09-05|Microsoft Technology Licensing, Llc|Predictive server-side rendering of scenes|
    GB2534847A|2015-01-28|2016-08-10|Sony Computer Entertainment Europe Ltd|Display|
    US9874932B2|2015-04-09|2018-01-23|Microsoft Technology Licensing, Llc|Avoidance of color breakup in late-stage re-projection|
    CN104834372A|2015-04-15|2015-08-12|程锦发|Multifunctional wearable human-computer interaction input equipment based on MEMS acceleration sensor|
    US10163180B2|2015-04-29|2018-12-25|Qualcomm Incorporated|Adaptive memory address scanning based on surface format for graphics processing|
    EP3295368A1|2015-05-13|2018-03-21|Google LLC|Deepstereo: learning to predict new views from real world imagery|
    US10810797B2|2015-05-22|2020-10-20|Otoy, Inc|Augmenting AR/VR displays with image projections|
    US9588598B2|2015-06-30|2017-03-07|Ariadne's Thread , Inc.|Efficient orientation estimation system using magnetic, angular rate, and gravity sensors|
    US10643381B2|2016-01-12|2020-05-05|Qualcomm Incorporated|Systems and methods for rendering multiple levels of detail|
    US9978180B2|2016-01-25|2018-05-22|Microsoft Technology Licensing, Llc|Frame projection for augmented reality environments|
    US10497089B2|2016-01-29|2019-12-03|Fotonation Limited|Convolutional neural network|
    US10317989B2|2016-03-13|2019-06-11|Logitech Europe S.A.|Transition between virtual and augmented reality|
    US10282806B2|2016-04-20|2019-05-07|Mediatek, Inc.|Graphics Accelerator|
    US11024014B2|2016-06-28|2021-06-01|Microsoft Technology Licensing, Llc|Sharp text rendering with reprojection|
    US10037626B2|2016-06-30|2018-07-31|Microsoft Technology Licensing, Llc|Interaction with virtual objects based on determined restrictions|
    US9978118B1|2017-01-25|2018-05-22|Microsoft Technology Licensing, Llc|No miss cache structure for real-time image transformations with data compression|
    US10514753B2|2017-03-27|2019-12-24|Microsoft Technology Licensing, Llc|Selectively applying reprojection processing to multi-layer scenes for optimizing late stage reprojection power|
    US10255891B2|2017-04-12|2019-04-09|Microsoft Technology Licensing, Llc|No miss cache structure for real-time image transformations with multiple LSR processing engines|US10962780B2|2015-10-26|2021-03-30|Microsoft Technology Licensing, Llc|Remote rendering for virtual images|
    US9978118B1|2017-01-25|2018-05-22|Microsoft Technology Licensing, Llc|No miss cache structure for real-time image transformations with data compression|
    US10242654B2|2017-01-25|2019-03-26|Microsoft Technology Licensing, Llc|No miss cache structure for real-time image transformations|
    US10514753B2|2017-03-27|2019-12-24|Microsoft Technology Licensing, Llc|Selectively applying reprojection processing to multi-layer scenes for optimizing late stage reprojection power|
    US10255891B2|2017-04-12|2019-04-09|Microsoft Technology Licensing, Llc|No miss cache structure for real-time image transformations with multiple LSR processing engines|
    US10939038B2|2017-04-24|2021-03-02|Intel Corporation|Object pre-encoding for 360-degree view for optimal quality and latency|
    US10649521B2|2017-04-24|2020-05-12|Intel Corporation|Compensating for high head movement in head-mounted displays|
    JP6711803B2|2017-10-13|2020-06-17|株式会社ソニー・インタラクティブエンタテインメント|Image generating apparatus and image generating method|
    US11100899B2|2019-08-13|2021-08-24|Facebook Technologies, Llc|Systems and methods for foveated rendering|
    KR102092735B1|2019-12-17|2020-03-24|주식회사 오썸피아|Method and apparatus of providing augemented reality contents to sightseeing telescope|
    WO2021182126A1|2020-03-09|2021-09-16|ソニーグループ株式会社|Information processing device, information processing method, and recording medium|
    WO2021226535A1|2020-05-08|2021-11-11|Qualcomm Incorporated|Multi-layer reprojection techniques for augmented reality|
    法律状态:
    2021-10-19| B350| Update of information on the portal [chapter 15.35 patent gazette]|
    优先权:
    申请号 | 申请日 | 专利标题
    US15/470,737|US10410349B2|2017-03-27|2017-03-27|Selective application of reprojection processing on layer sub-regions for optimizing late stage reprojection power|
    PCT/US2018/023212|WO2018183025A1|2017-03-27|2018-03-20|Selective application of reprojection processing on layer sub-regions for optimizing late stage reprojection power|
    [返回顶部]